Quality RTOS & Embedded Software

 Real time embedded FreeRTOS RSS feed 
Quick Start Supported MCUs PDF Books Trace Tools Ecosystem TCP & FAT




Loading

Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 14, 2014

Hi there, i'm just experiencing problems that freertos locks up in vListInsert, exactly at this line:

for( pxIterator = ( xListItem * ) &( pxList->xListEnd ); pxIterator->pxNext->xItemValue <= xValueOfInsertion; pxIterator = pxIterator->pxNext )

when calling xQueueReceive with portMAX_DELAY Theres only one line in a thread where i'm using the queue, and only one thread where I'm sending something to the queue (also only once).

I checked out the note directly above the freeze, nothing of this things caused the problem. I checked out the stacks, interupt priorities (i know the speciality of Cortex-M3 with priorities) and so on... Then i saw the comment above the NOTE:

/* If the list already contains a list item with the same item value then the new list item should be placed after it. This ensures that TCB's which are stored in ready lists (all of which have the same ulListItem value) get an equal share of the CPU. However, if the xItemValue is the same as the back marker the iteration loop below will not end. This means we need to guard against this by checking the value first and modifying the algorithm slightly if necessary. */

After checking out the values of xValueOfInsertion and pxNext->xItemValue, i discovered that both entries have the same value (its always "3"), and theres only one item in the list (pxList->uxNumberOfItems == 1). I just coudln't exactly figure out, when the xItemValue is set, and what the value it has means. Is it used in different cases? Sometimes something with priorities is assigned to it and sometimes a delay-value... What can i do to avoid the problem of 2 same xItemValues like described in the comment i found in the code? Anyone else having this trouble?

EDIT: Just checked out pointer-values in the pxIterator-struct. They seems correct, beacuse pxNext and pxPrevious have the same adress (adress is located im ram-region), this seems to be correct, cause there is only one item there.

Thx for responses. Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 14, 2014

I'm not sure I follow your post, but suspect the comments you are quoting are not related as they refer to the case where the value of the item being inserted is portMAXDELAY. portMAXDELAY is either 0xffffffff or 0xffff depending on whether you are using a 32, 16 or 8-bit architecture.

when calling xQueueReceive with portMAX_DELAY Theres only one line in a thread where i'm using the queue, and only one thread where I'm sending something to the queue (also only once).

Could you post the code that is both sending and receiving on the queue.

Then i saw the comment above the NOTE:

I think that comment is in list.c.

After checking out the values of xValueOfInsertion and pxNext->xItemValue, i discovered that both entries have the same value (its always "3")

As above, I think that comment is only related to the case where the value of the item being inserted is portMAX_DELAY.

I just coudln't exactly figure out, when the xItemValue is set, and what the value it has means

It depends on which list item it is the value for - in this case I suspect it represents the priority of the task so the tasks are blocked in priority order (it is actually configMAX_PRIORITIES minus the tasks priority).

Regards.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

Hi, thx for the fast response - I'll try to make my problem a little bit more clear.

Here's the code:

define configMAX_PRIORITIES 5

Task 1 has a Priority of 2 Task 2 has a Priority of 3

In task 1, a message is sent to task 2, and then waits for the response. >//Make sure response queue is empty >while(xQueueReceive(CANSDOResponseQueue, &retvalu32, 0) == pdTRUE); > >retvalu32 = CANSDOSTATESTARTTRANSFER; >xQueueSendToBack(CANSDOTransferQueue, &retvalu32, 0); //Send to Task 2 > >***xQueueReceive(CANSDOResponseQueue, &retvalu32, portMAX_DELAY) <-- Here, the lock happens***

This is where task 2 waits for work from task 1, after doing the work, it posts back into the CANSDOResponseQueue. >if(xQueueReceive(CANSDOTransferQueue, &MsgAnsweru32, 0) == pdTRUE) > //Do the work... > xQueueSendToBack(CANSDOResponseQueue, &MsgAnsweru32, 0); //No delay needed, queue should be empty anyway...

So, just to make my problem a little bit clearer: The lock happens exactly in this line in list.c - file: >for( pxIterator = ( xListItem * ) &( pxList->xListEnd ); pxIterator->pxNext->xItemValue <= xValueOfInsertion; pxIterator = pxIterator->pxNext ) > { > /* There is nothing to do here, we are just iterating to the > wanted insertion position. */ > }

In the list, theres only one item (pointed to by pxNext and pxPrevious) pxIterator->pxNext->xItemValue has a value of 3. xValueOfInsertion has a value of 3.

When I understand the loop correctly, this causes a lock-up because the abort-condition is never met: >pxIterator->pxNext->xItemValue <= xValueOfInsertion

When xItemValue is representing the priority (of what?), then everytime an item with same or higher priority should be placed in the list, it would cause a lock-up.

I just couldn't figure out what this list ist actualy used for? And what can i then do to avoid this lock-ups?

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 15, 2014

It looks like the list has already been corrupted because the last item in the list should always have an item value of 0xffffffff (assuming 32-bit). Therefore the loop will always terminate at some point (assuming no corruption) on the <= test because there should be something with an item value of 0xffffffff, and that code can only execute if the item value being inserted is less than 0xffffffff.

This type of corruption can often happen because of invalid interrupt priorities. Are you using any interrupts in your application (other than the RTOS interrupts)? For example, is you CAN peripheral interrupt driven?

What version of FreeRTOS are you using? Do you have configASSERT() defined?

Regards.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

Yes i use lot's of interrupts, it's just, the application itself is in testing for more than 2 years now, and i never experienced such an issue until now.

I'm using FreeRTOS V7.0.2. Actually I don't have configASSERT defined, but I'll checkit out now, thx for the hint!

Here's the list of Interrupt-Priorities and my config:

#define configKERNEL_INTERRUPT_PRIORITY 255
#define configMAX_SYSCALL_INTERRUPT_PRIORITY 48 // equivalent to 0x30, or priority 3.

#define USB_IRQ_PRIO 4 //USB
#define TIMER7_IRQ_PRIO 14 //USB Delay
#define DISP_MOSI_DMA_IRQ_PRIO 13 //LCD
#define DISP_TIM_IRQ_PRIO 13 //LCD
#define TIMER2_IRQ_PRIO 10 //RTOS Runtimestats
#define FOB_IRQ_PRIO 5 //FOB
#define FOB_TX_DMA_IRQ_PRIO 5 //FOB
#define FOB_TIMER_IRQ_PRIO 5 //FOB
#define ADC_DMA2_STREAM3_IRQ_PRIO 14 //ADC
#define ADC_OC_IRQ_PRIO 0 //ADC-Overcurrent --> must not be delayed The ISR does not call any FreeRTSO API functions
#define EFLASH_IRQ_PRIO 6
#define EEPROM_IRQ_PRIO 6
#define DSP_DMA_IRQ_PRIO 9 //DSP_COM
#define DSP_USART_IRQ_PRIO 9 //DSP_COM
#define DSP_TIMER_IRQ_PRIO 9 //DSP_COM

All interrupts until here worked for more than 2 years now without a problem

The CAN-Interrupts were added recently

#define CAN_RX0_IRQ_PRIO 4 //CAN-Com
#define CAN_RX1_IRQ_PRIO 5 //CAN-Com
#define CAN_TX_IRQ_PRIO 5 //CAN-Com
#define CAN_ERR_IRQ_PRIO 4 //CAN-Com

Since I'm using an STM32F205, I use the ST-standardlibrary to configure the interrupts, this looks like that:

NVIC_InitTypeDef NVIC_InitStructure;
NVIC_InitStructure.NVIC_IRQChannel = CAN1_RX0_IRQn;
NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = CAN_RX0_IRQ_PRIO;
NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0x0;
NVIC_InitStructure.NVIC_IRQChannelCmd = ENABLE;
NVIC_Init(&NVIC_InitStructure);

NVIC_PriorityGroupConfig(NVIC_PriorityGroup_4); // 4 bits for pre-emption priority 0 bits for subpriority.

I just did an additional testing with the last thing i changed: i use a mutex to block the transmission of CANMessages in a function CANTransmit(). CAN_Transmit doesn't free the mutex. To get a minimum time between 2 CAN-Messages, i start a Softwaretimer with 5ms Delay and the Callback-Function then calls xSemaphoreGive();.

void CAN_TxDelayTimerCallback(xTimerHandle pxTimer)
{
    //Senden wieder freigeben
    xSemaphoreGive(CAN_TxMutex);
}

When i remove the Software-Timer and give back the Mutex directly in the CAN_Transmit-Message, everything just works fine. Need to investigate this further.

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

Just tried out configASSERT like this:

#define configASSERT(x)     if(x == 0) vAssertCalled( __FILE__, __LINE__ )

void vAssertCalled( void *pfile, uint32_t line)
{
	while(1);
}

vAssertCalled is never called, although the lock-up occurs.

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 15, 2014

I'm using FreeRTOS V7.0.2. Actually I don't have configASSERT defined, but I'll checkit out now, thx for the hint!

That is a relatively old version. Later versions include additional configASSERT() calls to trap invalid interrupt priorities, but your version probably doesn't.

Here's the list of Interrupt-Priorities and my config:

That all looks fine.

I just did an additional testing with the last thing i changed: i use a mutex to block the transmission of CANMessages in a function CANTransmit(). CAN_Transmit doesn't free the mutex. To get a minimum time between 2 CAN-Messages, i start a Softwaretimer with 5ms Delay and the Callback-Function then calls xSemaphoreGive();.

void CANTxDelayTimerCallback(xTimerHandle pxTimer) { //Senden wieder freigeben xSemaphoreGive(CANTxMutex); }

I would have to dig out V7.0.2 to determine the effect of giving a mutex from a task other than the mutex holder. The behaviour has changed recently, so the latest version I have on my PC is not helpful in that regards.

Regards.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

I just realized, that it might have something to do with the priority of the Software-Timer. Previously the priority was set to 3. The lock-up occured almost everytime when i called CAN_Transmit() without delay in between. Like:

CAN_Transmit(FrameA);
//Task should be blocked on second call of CAN_Transmit, until Software-Timer-Callback is called and give back the Mutex.
CAN_Transmit(FrameB);

The CAN_Transmit-Function looks like this:

uint8_t CAN_TxMessage(frame)
{
    xSemaphoreTake(CAN_TxMutex, portMAX_DELAY);

    //sending frame...

    //If time couldn't be startet, free Mutex anyway
    if(xTimerStart(CAN_TxDelayTimer,0) != pdPASS)
    {
        xSemaphoreGive(CAN_TxMutex);
    }
    return pass;
}

After changing the priority to 5, the lockup still occurs, but much less regularly, just like once every 10 minutes or so. (the Frames are sent every 100ms).

I think i check out the newest version of FreeRTOS, it might be resolved by itself then...

Just wanted to say thx for the help so far!

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by heinbali01 on November 15, 2014

Hi David,

Just to make sure that you're looking at what you think you're looking at, a small patch:

~~~~~ -#define configASSERT(x) if(x == 0) vAssertCalled( FILE, LINE ) +#define configASSERT(x) if((x) == 0) vAssertCalled( FILE, LINE )

+volatile int errorcaught; + void vAssertCalled( void *pfile, uint32t line) { + portDISABLEINTERRUPTS( ); while(1) + { + // A debugger will stop here on a break + errorcaught++; + } } ~~~~~

  1. Before the while(1), try to disable further interrupts.
  2. A macro parameter shall be put between parentheses

ad 1) It has happened to me that one task was caught in vAssertCalled() while other tasks were still running.

ad 2) The parentheses will change the outcome:

~~~~~ configASSERT( A && B ); if( A && B == 0 ) // wrong if( ( A && B ) == 0 ) // ok ~~~~~

I assume you didn't make mistakes in the choice of the FromISR() versions:

~~~~~ xQueueSend() <> xQueueSendFromISR() xQueueReceive() <> xQueueReceiveFromISR() xSemaphoreGive() <> xSemaphoreGiveFromISR() xEventGroupSetBits() <> xEventGroupSetBitsFromISR() ... ~~~~~

Good luck, Hein


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 15, 2014

I just stepped through the following code in V7.0.2 and cannot see a reason why it would not be valid:

static void prvCallback( xTimerHandle xTimer )
{
  xSemaphoreGive( xMutex );
}
/*----------------------------------*/

static void prvCheckTask( void *pvParameters )
{
xTimerHandle xTimer;

  /* Just to remove compiler warning. */
  ( void ) pvParameters;
  xTimer = xTimerCreate( "timer", 
                         50, 
                         pdFALSE, 
                         NULL, 
                         prvCallback );

  /* Start with the mutex held. */
  xSemaphoreTake( xMutex, portMAX_DELAY );

  for( ;; )
  {
    /* Start the timer to give the mutex. */
    xTimerStart( xTimer, portMAX_DELAY );

    /* Take the semaphore, which will only be available
    when the timer gives it. */
    xSemaphoreTake( xMutex, portMAX_DELAY );
  }
}

Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

Hi Hein, thx for your patch - although the result keeps the same :-(

I checked out the FromISR() and normal Calls and i think they are all fine. What makes me suspicious, is that the lock doesn't occur every time the functions are called, sometimes it can be called several times befor lock occurs, and with a misplaced xQueue... this wouldn't be the case?

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

What can be a reason for the list to be corrupted ? Except of those things can be found in the note 1) Stack overflow 2) Incorrect interrupt priority assignment, especially on Cortex-M parts where numerically high priority values denote low actual interrupt priorities, which can seem counter intuitive. 3) Calling an API function from within a critical section or when the scheduler is suspended, or calling an API function that does not end in "FromISR" from an interrupt. 4) Using a queue or semaphore before it has been initialised or before the scheduler has been started (are interrupts firing before TaskStartScheduler() has been called?).

It's just to make it easier for me, that i know what to look for in the code.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 15, 2014

The usual suspects are in the code comments you have already quoted. I presume you have also read through this: http://www.freertos.org/FAQHelp.html

Of course any misbehaving code could just be writing over the data structures.

Which heapn.c file are you using? If heap3.c could it be your stack and heap crashing into each other?

Regards.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

Uhm, yeah i guess i just solved the problem. I updated to the newest Version of FreeRTOS and the configASSERT just gave me the solution. I called xTimerIsTimerActive() from withing an ISR. I thought, when implementing this, that it's ok because there is no FromISR port of this function and there's no prohibition in API reference.

Just worked around with setting a variable and now it just works fine.

Is there a possibility to check out if the timer is active from within an ISR?

Sorry guys for consuming so much time from you with such a stupid mistake made by me! Thank you so much for your help!

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 15, 2014

Completely untested, but try adding the following into timers.c:


BaseType_t xTimerIsTimerActiveFromISR( TimerHandle_t xTimer )
{
BaseType_t xTimerIsInActiveList;
Timer_t *pxTimer = ( Timer_t * ) xTimer;
UBaseType_t uxSavedInterruptStatus;

  /* Is the timer in the list of active timers? */
  uxSavedInterruptStatus = portSET_INTERRUPT_MASK_FROM_ISR();
  {
    /* Checking to see if it is in the NULL list in effect checks to see if
    it is referenced from either the current or the overflow timer lists in
    one go, but the logic has to be reversed, hence the '!'. */
    xTimerIsInActiveList = ( BaseType_t ) !( listIS_CONTAINED_WITHIN( NULL, &( pxTimer->xTimerListItem ) ) );
  }
  portCLEAR_INTERRUPT_MASK_FROM_ISR( uxSavedInterruptStatus );

  return xTimerIsInActiveList;
}

Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 15, 2014

ah sh... Just realized, that this didn't solve problem. During the change to new FreeRTOS-Version i expirienced a bluescreen on PC. This bluescreen somehow, compromised my projectsettings so that some Symbols were resetted. I needed this symbols so simulate a proper CAN-Communication, so the code having the problem, wasn't even executed during my test with the new version of FreeRTOS.

The lock still happens with the new and the old version of FreeRTOS, altough, with the newer Version it even ran 10 Minutes without a mistake, just to freeze then.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 17, 2014

Hi again!

After some additional testing, i think that i've tracked down the problem to the priority switching of a mutex-owner due to a blocked mutex.

Crash occures with following behaviour:

Task 1 with priority 2. Task 2 with priority 3. TimerTask with priority 3.

01: Task 1 takes TxMutex. 02: Task 1 starts Timer 5ms. 03: Task 1 is blocked in xQueueReceive(ResponseQueue), waiting for message from Task 2. 04: Task 2 wants to take TxMutex--> Is blocked 05: Task 1 then should be set to priority 3, because he own's TxMutex 06: TimerTask gives TxMutex 07: Task 2 finally takes TxMutex 08: Task 2 starts TimerTask 5ms 09: Task 2 sends into ResponseQueue of Task 1 10: TimerTask gives TxMutex 11: Task 1 receives Message from Task 2 in ResponseQueue 12: Task 1 does some work... 13: Task 1 calls xQueueReceive(ResponseQueue) again 14: Task 1 crashes in list.c file due do corrupt list.

After i detected this, I changed the mutex to a semaphore, to get the priority change out of list. Crash disappeared, everything is working fine right now.

Might there be a problem with the priority-change? When a mutex-owner is blocked and it's priority is changed or something like that?

Additional Info to the previous post: It seem's that with newest version, the problem is also fixed. Couldn't reproduce the crash with newest version.

Best regards, David


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 17, 2014

06: TimerTask gives TxMutex

...but the timer task was not the mutex holder.

Additional Info to the previous post: It seem's that with newest version, the problem is also fixed. Couldn't reproduce the crash with newest version.

That is alright then ;o) All the same when I get the chance I will play through this scenario to make sure it is safe.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by deiti123 on November 17, 2014

...but the timer task was not the mutex holder.

Does this mean that a mutex has to be given back only from the task who's the current owner of the mutex?

I also checked out another scenario where 01: Task 1 takes TxMutex 02: Task 1 starts Timer 5ms were skipped, this also works perfectly (although timertask still give back mutex)

Only difference here, is that there aren't 2 different tasks who want to take the mutex.


Crash in vListInsert due to xValueOfInsertion is equal to pxNext->xItemValue

Posted by rtel on November 17, 2014

I would have to walk through the scenarios in a particular version. This was changed recently to allow easier integration with third party code (that was fixed, so could not be edited to match our model).

The special case with the mutex is the pointer to the mutex holder. This is why, for example, [in the current version] a mutex can be given from an interrupt but cannot be taken from an interrupt. Not that it would make sense for an interrupt to hold a mutex anyway, but if it were to attempt to hold one there would be no persistent 'mutex holder'.


[ Back to the top ]    [ About FreeRTOS ]    [ Sitemap ]    [ ]




Copyright (C) Amazon Web Services, Inc. or its affiliates. All rights reserved.

Latest News

FreeRTOS kernel V10 is available for immediate download. Now MIT licensed.


FreeRTOS Partners

ARM Connected RTOS partner for all ARM microcontroller cores

IAR Partner

Microchip Premier RTOS Partner

RTOS partner of NXP for all NXP ARM microcontrollers

STMicro RTOS partner supporting ARM7, ARM Cortex-M3, ARM Cortex-M4 and ARM Cortex-M0

Texas Instruments MCU Developer Network RTOS partner for ARM and MSP430 microcontrollers

OpenRTOS and SafeRTOS