Quality RTOS & Embedded Software

 Real time embedded FreeRTOS RSS feed 
Quick Start Supported MCUs PDF Books Trace Tools Ecosystem TCP & FAT




Loading

Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 30, 2016

Hi, my system has sporadic hard faults, this means, working over night, or crashes within minutes and all between. I use the suggested code:

void prvGetRegistersFromStack( uint32t pulFaultStackAddress ) { / These are volatile to try and prevent the compiler/linker optimising them away as the variables never actually get used. If the debugger won't show the values of the variables, make them global my moving their declaration outside of this function. */ volatile uint32t r0; volatile uint32t r1; volatile uint32t r2; volatile uint32t r3; volatile uint32t r12; volatile uint32t lr; /* Link register. */ volatile uint32t pc; /* Program counter. / volatile uint32_t psr;/ Program status register. */

r0 = pulFaultStackAddress[ 0 ];
r1 = pulFaultStackAddress[ 1 ];
r2 = pulFaultStackAddress[ 2 ];
r3 = pulFaultStackAddress[ 3 ];

r12 = pulFaultStackAddress[ 4 ];
lr = pulFaultStackAddress[ 5 ];
pc = pulFaultStackAddress[ 6 ];
psr = pulFaultStackAddress[ 7 ];

/* When the following line is hit, the variables contain the register values. */
for( ;; );

}

Now I need some help to analyse the reason for the crash. All registers except psr were 0. Even pc is 0. Any idea how to get closer to the problem?

Thanks again for helping.

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by davedoors on January 30, 2016

Doesnt sound right. Can you show the code that calls prvGetRegistersFromStack() too. You might need to dig deeper into the Cortex system registers to see what caused the hard fault (memory access, what ever). You could also use the trace tool with a circular trace buffer that is stopped in the fault handler.


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 30, 2016

Thanks, any documents around to set up the trace tool with such a buffer configuration ?

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 30, 2016

Hi, ok, tried to follow the instructions in this document, but it ends up getting 93 errors ... http://www.atmel.com/images/atmel-42382-getting-started-with-freertos-on-atmel-sam-flash-mcusapplicationnoteat04056.pdf

Maybe there is a version conflict, as the files having different names. In my source of the trace recorder I dont have a file called trcHooks.h, but I have trcKernelHooks.h.

I cut it out now, so my project compiles again,

Any suggestion how to get it running. Which port should I use for a sam4N, I choose ARM Cortex-M, maybe thats the mistake?

Thanks for helping

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

Any suggestion how to get it running.

If you are using Atmel Studio, then you may find the ASF already has a simple FreeRTOS project for that part. Try using the ASF wizard, select your part, and then see which projects are provided for that part (there is likely to be a lot, a couple of which are likely to be FreeRTOS projects).

Otherwise the following links might be helpful: http://www.freertos.org/Creating-a-new-FreeRTOS-project.html http://www.freertos.org/porting-a-freertos-demo-to-different-hardware.html

Wich port should I use for a sam4N, I choose ARM Curtex-M, maybe thats the mistake?

With any Cortex-M4 microcontroller, if the microcontroller has a floating point unit (FPU) then use the code from FreeRTOS/Source/portable/[compiler]/ARMCM4F, otherwise use the code from FreeRTOS/Source/portable/[compiler]/ARMCM3.


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

Hi, thanks. But, I have a running application for my SAM4N and I use Atmel Studio V6.2. Now I want to add the trace recorder, but this wouldn't work out .... In the file trcConfig.h I am ordered to enter a port option, here I choose Cortex-M, hoping this is the correct port (of the trace recorder) for my SAM4N. The port of the FreeRTOS 7.3.0 I still use works correct for my app, nothing to change for now.

Its only about adding the trace recoder to an exixting application.

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

Hi, to be more precise:

First problem appear in trcHardwarePort.c. In the function prvTraceInitCortexM, near the end the statements:

if (RecorderDataPtr->frequency == 0) { RecorderDataPtr->frequency = TRACECPUCLOCKHZ / HWTCDIVISOR; }

fail, cause the macros used here lastly end up calling 'sysclkgetcpu_hz'

Problem is now, which file where to include to solve this and not produce lots of duplicate definitions elswhere ....

It sounds like a basic issue, maybe the document describing the integration of the trace recorder lib should be updated to reflect the needs of the current version.

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

This looks Atmel specific. I'm guessing TRACECPUCLOCKHZ is somewhere referencing configCPUCLOCKHZ (which is in FreeRTOSConfig.h), and configCPUCLOCKHZ is calling sysclkgetcpuhz (which I think is an Atmel library function) rather than just hard coding the clock speed.

If you right click on TRACECPUCLOCKHZ you should find its definition, and follow it back to the base definition. I'm guessing replacing sysclkgetcpuhz() with the actual clock speed as a hard coded value will fix the issue.


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

.. I actually trace this back as you suggested: trcHardwarePort.c references trcKernelPort.h:

~~~~

if (USETRACEALYZERRECORDER == 1)

/* Defines that must be set for the recorder to work properly */

define TRACEKERNELVERSION 0x1AA1
define TRACETICKRATEHZ configTICKRATE_HZ /* Defined in "FreeRTOS.h" */
define TRACECPUCLOCKHZ configCPUCLOCK_HZ /* Defined in "FreeRTOSConfig.h" */
if (SELECTEDPORT == PORTARM_CortexM)

~~~~ And in FreeRTOS.h I have the definitions as normal. Doesn't changed anything here. Without trace recorder, my FreeRTOS based application works fine.

Here the macros:

~~~~

define configUSE_PREEMPTION 1
define configUSEIDLEHOOK 0
define configUSETICKHOOK 0
define configCPUCLOCKHZ ( sysclkgetcpu_hz() )
define configTICKRATEHZ ( ( portTickType ) 1000 )
define configMAXPRIORITIES ( ( unsigned portBASETYPE ) 5 )
define configMINIMALSTACKSIZE ( ( unsigned short ) 150 )

~~~~

If in FreeRTOS.h there were a mistake, my app would not compile, is this correct?

Ok, maybe keying in the CPU frequency for TRACECPUCLK_HZ would help, the ... DIVISOR also use this macro ...

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

This is what I suspected:

define configCPUCLOCKHZ ( sysclkgetcpu_hz() )

Did you try replacing the call to sysclkgetcpu_hz() with the hard coded clock frequency, as just suggested?


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

Yes, but now other issues popping up ... endless strory Has anybody tried it before, look and feel is, that this hasn't worked out before ...

Now the statement '' include "trcKernelHooks.h" informed me, that this file isn't available ???? I still can see the file. And with right click and 'Go to implementation' I get exactly there.

When I use an absolute path, the error is gone and I get 97 errors telling me about lots of duplicated definitions. So, better to exclude this file? But not included gives also an error. Really great stuff.

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

I add a shot. I exactly tried as in this document is suggested: http://www.atmel.com/images/atmel-42382-getting-started-with-freertos-on-atmel-sam-flash-mcusapplicationnoteat04056.pdf

Doing both, adding a folder as you see in the attachment and adding a search path solves some how the duplicate definitions, but not every time. Depends which of the statements fail and which not.

It is a lottery ...

With best regards

Gerhard

Attachments


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

Has anybody tried it before, look and feel is, that this hasn't worked out before

Um, "yes", a few hundred if not thousands of people, me included.

Attached you will find what I am using on an EFM32 - which is also a Cortex-M, so should be the same. The zip file has the files and path I have in the project, and the header file is the config file. These are from a training I did before Christmas, so might not be the latest version.

Attachments

FreeRTOS%2BTrace.7z (43479 bytes)
trcConfig.h (22661 bytes)

Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

Thanks, will see where is the difference. I have to use FreeRTOS 7.3.0, so far from the latest version, cause Atmel didn't support an other version on SAM4N MCU.

Hope there is a way to have a 'TRACE' version and a DEBUG version of my project, have to work on the app not playing around with trace integration. Uptil now I didn't need a trace, but once a day it would be helpful. Especially as I want to find the sopradic crashes, see start of this discussion. But if integration is that hard ...

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

.. where should I place the trcConfig.h file. I have a Config folder where the FreeRTOSConfig.h resides and some more, is this the exact place for it ? (I guess .. but ...)

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by alainm3 on January 31, 2016

I am following this and I have a different experience !!!

Up to now, ALL my sporadic hardfaults happend because of stack overflow (CM3 and yes, I have been lucky..). I included in the system a stack report function, almost exactly as exists in the Examples sections and now I can diagnose that in seconds.

I tested the stack detection feature that exists in FreeRTOS specific tests and I discovered that it failed to avoid the hard fault in most tested situations...

Alain

On 31-01-2016 13:51, Real Time Engineers ltd. wrote: > > Has anybody tried it before, look and feel is, that this hasn't > worked out before > > Um, "yes", a few hundred if not thousands of people, me included. > > Attached you will find what I am using on an EFM32 - which is also a > Cortex-M, so should be the same. The zip file has the files and path I > have in the project, and the header file is the config file. These are > from a training I did before Christmas, so might not be the latest > version. > > Attachments: > > * FreeRTOS+Trace.7z > https://sourceforge.net/p/freertos/discussion/382005/thread/53b486f2/6592/attachment/FreeRTOS%2BTrace.7z > (43.5 kB; application/octet-stream) > * trcConfig.h > https://sourceforge.net/p/freertos/discussion/382005/thread/53b486f2/6592/attachment/trcConfig.h > (22.7 kB; text/plain) > > ------------------------------------------------------------------------ > > Sporadic hard faults, how to find reason > https://sourceforge.net/p/freertos/discussion/382005/thread/53b486f2/?limit=25#6592 > > ------------------------------------------------------------------------ > > Sent from sourceforge.net because you indicated interest in > https://sourceforge.net/p/freertos/discussion/382005/ > > To unsubscribe from further messages, please visit > https://sourceforge.net/auth/subscriptions/ >

Attachments

alternate (2870 bytes)

Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

.. same situation.

It is now a miracle for me, how to integrate trace into a project.

I alwas get the error, that Atmel Studio can't find 'trcKernelHooks.h'.

Lastly I tried, to type #include "", than tab and he showed me a list of files. HE shows me the list and I think, he will find this files ... I selected trcKernelHooks.h, and still get the ... file not found error ..

I also restarted the Atmel Studio 6.2, nothing helps.

Just a miracle.

Is it possible to add trace later or should I have added it at the beginning and now it is impossible for whar reason ever?

No more idea .... sorry

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

.. ok, now hanging at:

Error 34 implicit declaration of function '_getPRIMASK' [-Werror=implicit-function-declaration] .......srcTraceRecordertrcBase.c 432 2


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

Putting _getPRIMASK into Google shows me that this is a CMSIS function. Do you have the CMSIS headers included?


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

No, there is no info about that. But the ASF have such files added. Ok, will try around


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

... oh, this include in trcConfig.h helps:

~~~~ /* Silicon Labs includes. */ //#include "em_chip.h"

include "armconststructs.h"

/****************************************************************************** * SELECTED_PORT * ~~~~

and there is a problem within trcHardwarePort.c, I need to hard code the processors clock frequency ..

~~~~ if (RecorderDataPtr->frequency == 0) { // RecorderDataPtr->frequency = TRACECPUCLOCKHZ / HWTCDIVISOR; RecorderDataPtr->frequency = 100000000 / HWTC_DIVISOR; } ~~~~

Doesn't kow if it really works but it compiles now, this is a lot ..

And now the big question: Hoe can I have a 'TRACE' configuration and a 'NOTRACE' configuration. like the 'Degug' and 'Release' configurations in Studio 6.2

Thanks a lot

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

... and what do I get after hours of fiddling around?

As soon as I call 'uiTraceStart();' I get an hard fault ... really nice ....


Sporadic hard faults, how to find reason

Posted by rtel on January 31, 2016

And now the big question: Hoe can I have a 'TRACE' configuration and a 'NOTRACE' configuration. like the 'Degug' and 'Release' configurations in Studio 6.2

Either include the header file at the bottom of the FreeRTOSconfig.h file, or don't. If you don't include it, also don't initialise or start the trace recorder.

As soon as I call 'uiTraceStart();' I get an hard fault ... really nice ....

Did you call vTraceInitTraceData() before calling vTraceStart()?


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

... ok, some step further.

The Atmel document is defenitely wrong, they try to tell me, I only have to invoke uiTraceStart(); Correct is that I first have to invoke

~~~~ vTraceInitTraceData(); uiTraceStart(); ~~~~

So, now I tried to setup the free version of the viewer. Ok, I found the infos in the map file, it looks different than described, but I am pretty shure I got it.

Now they tell me, that I have input this data in the 'Settings --' menue. But, this, and all others were disabled .... And there is a menu point 'Atmel Studio', here I am able to click 'Read', but .... Error.

Ok, and now?

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on January 31, 2016

.. ok, second attempt works, works? I see a red trace, named Startup, no of my tasks, semaphors, nothing .... just useless.

Where can I find a document telling me how to connect this tool to my Atmel Studo or HW so I really get useful information ....

With best regards

Gerhard


Sporadic hard faults, how to find reason

Posted by rtel on February 1, 2016

I think the stand-alone trace viewer/IDE has a help system, accessible by the help menu, that has quite a lot of useful documentation.


Sporadic hard faults, how to find reason

Posted by gerhardkreuzer on February 2, 2016

.. works. Problem was installing the hooks .... Case can be closed now.


[ Back to the top ]    [ About FreeRTOS ]    [ Sitemap ]    [ ]




Copyright (C) Amazon Web Services, Inc. or its affiliates. All rights reserved.

Latest News

FreeRTOS kernel V10 is available for immediate download. Now MIT licensed.


FreeRTOS Partners

ARM Connected RTOS partner for all ARM microcontroller cores

IAR Partner

Microchip Premier RTOS Partner

RTOS partner of NXP for all NXP ARM microcontrollers

STMicro RTOS partner supporting ARM7, ARM Cortex-M3, ARM Cortex-M4 and ARM Cortex-M0

Texas Instruments MCU Developer Network RTOS partner for ARM and MSP430 microcontrollers

OpenRTOS and SafeRTOS